It is best to keep lithium batteries charged. However, that may not be the problem.
The possibilities are USB port not working, faulty USB cable, or hardware problem on shuffle.
To rule out the USB port, connect something else there, preferably something like the shuffle, such as a USB flash drive.
To rule out a faulty USB cable, you'll need to try a different one or try that one with another shuffle. Is the cable you are using the original short cable? Some third-party replacements are for the 2nd gen shuffle; they will not work with the 4th (or 3rd) gen shuffle, because the connector is different.
If you confirm the cable is good, and the USB port is working, the cause must be a hardware problem on the shuffle.